On the maximum entropy of the sum of two dependent random variables

If pi(i = l , * . ,N)is the probability of the ith letter of amemoryless souree, the length li of the e o r r e s p d i n g binary HufFmancodeword can be very different from the d u e -logpi. For atypical letter, however, Ii = -log pi. Mote precisely, Pm=E {ill o g p -m)Pj < 2-" and P,'= zjE{il,,>-logp,+m)Pj <2-c(m15T1, &re; c = 227. Index Term.s-Huffman code, length of a typical codeword. I. RESULTSConsider a discrete memoryless N-letter source ( N 2 2) towhich a binary Huffman code [l] is assigned. The ith letter hasprobability p , < 1 and codeword length 1,. For a dyadic source(i.e., allp , are negative powers of 2) the Huffman codewordlengths 1, are equal to the self information, -log p,, for all i .More general sources, however, may give rise to atypical lettersfor which the Huffman codeword lengths differ greatly from theself information. Given p, , the length 1, can in principle be assmall as 1 and as large as {log[(fi + 1)/2]}-'(-10g p , ) =1.44(-log p , ) El.In this correspondence, bounds on the probability of suchatypical letters are derived. It is shown that the probability ofthe letters for which the Huffman codeword length differs bymore thanm bits from -log p , decreases exponentially with m.In this sense, one can say that the Huffman codeword for atypical letter satisfies 1, = -log pi . This result has an applicationto recent fundamental questions in statistical physics [31, [41.The Huffman code can be represented by a binary tree havingthe siblingproperty [5] defined as follows. The number of linksleading from the root of the tree to a node is called theleuel ofthat node. If the level-n node a is connected to the level&+ 1)nodes b and c, then a is called the parent of b and c; a'schildren b and c are called siblings. There are exactly N terminalnodes or leuues, each leaf corresponding to a letter. Each linkconnecting two nodes is labeled 0 or 1. The sequence of labelsencounteredon the path from the root to a leaf is the codewordassigned to the corresponding letter. The codeword length of aletter is thus equal to the level of the corresponding leaf. Eachnode is assigned a probability such that the probability of a leafis equal to the probabilityof the corresponding letter and theprobability of each nonterminal node is equal to the sum of theprobabilities of its children. A tree has the sibling property ifand only if each node except the root has a sibling and the nodescan be listed in order of nonincreasing probability with eachnode being adjacent to its sibling in the list [5].Definition: A level-1 node with probability p-or, equivalently,a letter with probability p and codeword length 1-has thepropertyXL(X;)ifandonlyifl> l o g p + m ( l < -1ogp-m).Theorem 1: P i = C,,,p, < 2-"' where I; = {ill, <-log p , m), i.e., the probabdity that a letter has property X;is smaller than 2-". (This is true for any prefix-free code.) Manuscript received January 27, 1993; revised August 27, 1993. Thiswork of the author was supported by a fellowship from the DeutscheForschungsgemeinschaft.The author is with the Department of Physics and Astronomy, Univer-sity of New Mexico, Albuquerque, NM 87131.IEEE Log Number 9403832. T0018-9448/94$04.00 6 1994 JEEE